AD-A284  573 

1  llllltl  ill*  nail  . . . 


CDRL:  B019 
31  March  1994 


imsYS 


Metrics  Concept  Report 

Central  Archive  for  Reusable  Defense  Software 
(C\RDS) 


aA-29953 


imc  QJALiii  iiJoPECTED  a 


CDRL:  B019 
31  March  1994 


INFORMAL  TECHNICAL  REPORT 
For  The 

SOFTWARE  TECHNOLOGY  FOR  ADAPTABLE,  RELIABLE  SYSTEMS 

(STARS) 


Metrics  Concept  Report 
Central  Archive  for  Reusable  Defense  Software 
(CARDS) 


STARS-VC-B019/004/00 
31  March  1994 

Infonnal  Technical  Data 

CONTRACT  NO.  F19628-93-C-0130 
Line  It^  (XX)2AB 

Prcparedfor: 

Electronic  Systems  Center 
Air  Force  Materiel  Command,  USAF 
Hanscom  AFB,  MA  01731-2116 

Prepared  By: 

HGO  Technology,  Inc. 
under  contract  to 
Unisys  Coiporation 
12010  Sunrise  Valley  Drive 
Reston  VA  22091 


Distribution  Statement  ‘‘A” 
per  DoD  Directive  5230 

Approved  for  public  release,  distribution  is  unlimited 


CDRL:  B019 
31Marchy  1994 


INFORMAL  TECHNICAL  REPORT 
For  The 

SOFTWARE  TECHNOLOGY  FOR  ADAPTABLE,  RELUBLE  SYSTEMS 

(STARS) 


Metrics  Concept  Report 
Central  Archive  for  Reusable  Defense  Software 
(CARDS) 


STARS-VC-B019/004y00 

Infonnal  Ibchnical  Data 
31  March  1994 

CONTRACT  NO.  F19628-93-C-0130 
Line  Item  0(X)2AB 

Prepared  for; 

Electronic  Systems  Center 
Air  Force  Material  Command,  USAP 
Hanscom  AFB,  MA  01731-2116 


CDRL:  B019 
31  March  1994 

Data  Reference:  STARS-VC-B019/004/00 
INFORMAL  TECHNICAL  REPORT 
Metrics  Concept  Report 

Central  Archive  for  Reusable  Defease  Software  (CARDS) 

DlitribuUon  Statamint  ‘‘A” 
perDoDDIractivb5Z30J4 

Approved  fbr  public  nlMMi  Dlitributlon  li  unlimited. 

Ccq^yright  1994,Unisys  Cenporation,  Reston,  \%ginia 
and  HOO  Ibctoology,  Inc. 

Copyright  is  assigned  to  the  U.  S.  Government,  upon  delivery  thereto,  in  accordance  with 

the  DEARS  Special  Works  Clause. 

This  document,  developed  under  the  Software  Ibchnology  for  Adiq)table,  Reliable  Systems 
(STARS)  program,  is  approved  for  release  under  Distribution  ‘‘C**  of  the  Sdentifle  a^  Technical 
Information  Program  Ossification  Scheme  (DoD  Directive  S230.24)  unless  otherwise  indicated. 
Sponsored  by  the  U.  S.  Advanced  Research  Rejects  Agency  (ARPA)  under  contract  P19628-93* 
C>0130  the  STARS  program  is  supported  by  the  military  services.  SEI,  and  MURE,  with  the 
U.  S.  Air  Force  as  the  executive  contracting  agent.  The  information  identified  herein  is  subject  to 
change.  For  further  information,  contact  the  authors  at  the  following  mailer  address: 
delivery@star8jeston.paramax.com. 

Permission  to  use,  copy,  modify,  and  comment  on  this  document  for  purposes  stated  under  Distri¬ 
bution  **C”  and  without  fee  is  hereby  granted,  providing  that  this  notice  appears  in  each  whole  or 
partial  copy.  This  document  retains  Contractor  indemnification  to  the  Government  regarding 
copyrights  pursuant  to  the  above  referenced  STARS  contract.  The  Government  disdaims  all 
responsibility  against  liability,  induding  costs  and  expenses  for  violation  of  property  rights,  or 
copyrights  arising  out  of  the  creation  or  use  of  this  document. 

The  contents  of  this  document  constitutes  technical  information  developed  for  internal  Govern¬ 
ment  use.  The  Government  does  not  guarantee  the  accuracy  of  the  contents  and  does  not  sponsor 
the  release  to  third  parties  whether  engaged  in  performance  of  a  Government  contract  or  subcon¬ 
tract  or  otherwise.  The  Government  further  disallows  any  liability  fur  damages  incurred  os  the 
result  of  the  dissemination  of  this  information. 

In  addition,  the  Government  (prime  contractor  or  its  subcontractor)  disclaim  all  warranties  with 
regard  to  this  document,  including  all  implied  warranties  of  merchantability  and  fitness,  and  in  no 
event  shall  the  Government  (prime  contractor  or  its  subcontractor)  be  liable  for  any  special,  indi¬ 
rect,  or  consequential  damages  or  any  damages  whatsoever  resulting  from  the  loss  of  use,  data,  or 
profits,  whether  in  action  of  the  contract,  negligence,  or  other  tortious  action,  arising  in  connec¬ 
tion  with  the  use  or  performance  of  his  document. 


CDRL:  B019 
31  March  1994 

INFORMAL  TECHNICAL  DOCUMENT 

Metrics  Concept  Report 

Central  Archive  for  Reusable  Defense  Software 

(CARDS) 


Princ4>al  author: 


Anita  Berns 


Approvals; 


System  Architect  WaUnau 


Program  Manager  Lorraine  Martin 


(signatures  on  Fils) 


CDRL:  B019 
31  March  1994 

Data  Reference:  STARS- VC-B019/004/00 
INFORMAL  TECHNICAL  REPORT 
Metrics  Concept  Report 

ABSTRACT 

In  February  1993  the  Defense  Information  Systems  Agency/Center  for  Informa¬ 
tion  Management  (DISA/CIM)  sponsored  a  Software  Reuse  Metrics  Workshcp. 

The  primary  goal  of  these  meetings  was  to  develop  a  set  of  questions  that  would 
indicate  what  measurements  should  be  collected  in  support  of  the  DISA/CIM 
Software  Reuse  Program.  The  results  of  this  workshop  are  published  in  the  DIS- 
A/QM  Software  Reuse  Program  Reuse  Metrics  Workshop  Proceedings,  23-24 
February  1993  [DISA93]. 

These  proceedings  indicate  that  DISA/CIM*s  Software  Reuse  Metrics  Plan  will 
be  used  as  a  foundation  for  developing  a  DoD  Software  Reuse  Metrics  Plan.  The 
DISA/CIM  plan  serves  as  a  major  input  into  the  DoD-wide  plan,  and  DIS¬ 
A/CIM  will  be  conducting  pile*  prototypes  of  their  recommendations. 

The  CARDS  (Central  Archive  for  Reusable  Defense  Software)  metrics  effortss- 
hould  also  provide  valuable  information  that  can  be  feed  into  the  DoD-wide 
Software  Reuse  Metrics  Plan.  As  stated  in  the  DoD  Software  Vision  and  Strat¬ 
egy  Document,  the  DoD  must  establish  a  baseline  upon  which  to  gauge  success 
and  measure  improvement  that  serve  as  a  basis  for  comparison  among  alterna¬ 
tive  approaches. 

Therefore,  the  CARDS  metrics  effort  is  two-fold:  (1)  use  metrics  within  the 
CARDS  Program  to  measure  and  improve  processes,  products,  and  services  and 
(2)  monitor  and  provide  po'isible  contributions  to  the  DoD  Software  Reuse  Met¬ 
rics  Plan.  The  Metrics  Concept  Report  focuses  mostly  on  metrics  within 
CARDS.  As  experience  is  gained  within  the  CARDS  Program,  more  effort  will 
be  focused  towards  the  development  of  input  for  the  DoD  Software  Reuse  Met¬ 
rics  Plan.  However,  decisions  will  be  made  in  support  of  the  larger  DoD-wide 
perspective. 
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Metrics  Concepts  Report 

1  Purpose  and  Intended  Audience  of  the  Metrics 
Concepts  Report 

This  document  provides  a  framework  for  defining  and  implementing  a  measurement  process  for  the 
CARDS  Program,  its  processes,  products,  and  services. 

The  intended  audience  of  this  document  is  the  CARDS  Program  Manager,  System  Architect,  Franchise 
Coordinator,  Project  Leads,  and  all  other  CARDS  team  members.  Readers  outside  the  CARDS  organi* 
zation  should  be  familiar  with  the  CARDS  Program,  its  terminology  and  organhation.  Suggested  read¬ 
ings  are  listed  in  the  reference  section. 

1.1  Structure  of  this  Document 

The  Introduction  Section  describes  how  metrics  flow  within  the  CARDS  Program,  outlines  the  goals  of 
the  CARDS  measurement  process,  and  briefly  describes  Victor  Basili*s  Ooal/Question/Metric  (GQM) 
paradigm. 

The  CARDS  Environment  Section  focuses  on  the  challenges  produced  by  the  CARDS  environment:  the 
organizational  structure,  the  CARDS  Program,  the  implementation  of  franchises,  and  the  readiness  of 
the  CARDS  organization  for  metrics. 

The  Measurement  Process  Section  describes  a  six-step  process  and  a  matrix  framework  that  will  be 
used  to  relate  metrics  back  to  their  originatiog  questions,  issues,  and/or  goals. 

The  Metrics  Definition  Phase  Section  explains  the  first  three  stq;>s  of  the  six-step  process;  defining 
goals,  defining  processes,  and  defining  metric  questions  and  metrics. 

The  Metrics  Implementation  Phase  Section  describes  the  last  three  steps  of  the  six-step  process;  collect¬ 
ing  the  metrics,  analyzing  the  metrics,  and  acting  on  the  results  of  the  analysis. 

The  Added  Value  through  Automation  Section  describes  a  database  tool  to  document  the  metrics  and 
facilitate  analysis. 

1.2  Relationship  to  Other  CARDS  Documents 

The  measurement  process  will  first  be  applied  to  the  CARDS  Command  Center  Library  (CCL)  and  doc¬ 
umented  in  a  Metrics  Plan.  Lessons  learned  will  be  used  to  refine  the  measurement  process  before  pos¬ 
sibly  expanding  it  to  other  CARDS  project  areas  (domain  engineering,  franchise  implementation, 
franchise  concepts,  training,  and  reuse  coordination). 

All  metrics  definition,  collection,  analysis,  and  presentation  procedures  created  will  be  documented  in 
the  CARDS  Library  Operation  Policies  and  Procedures  Manual  (LOPP). 
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2  Introduction 

A  metric  is  a  characteristic  (such  as  lines  of  code,  number  of  defects,  or  defects/Unes  of  code)  of  a  pro¬ 
cess,  service,  or  product.  Metrics  are  useful  for  indicating  where  a  process,  service,  or  product  can  be 
improved,  or  when  a  goal  has  been  met  The  definition,  collection,  and  analysis  of  metrics  is  a  cyclical 
process.  Lessons  learned  fiom  metrics  information  can  lead  to  modifications  in  the  CARDS  Program 
and  project  goals,  which  may  result  in  refinement  of  the  processes.  For  a  measurement  process  to  pro¬ 
vide  benefits  to  a  project  the  definition,  collection,  and  analysis  of  mettles  must  be  guided  by  carefully 
identifying  appropriate  goals. 

The  metrics  addressed  in  this  Metrics  Concept  Report  are  closely  aligned  with  the  principles  of  Total 
Quality  Management  CTQM)  and  Continuous  Process  Improvement.  This  chapter  describes  the  flow  of 
metrics  information  within  CARDS,  lists  the  measuiement  goals  for  CARDS,  and  briefly  describes  the 
OQM  paradigm  as  defined  by  Victor  Basili  for  systematically  eliciting  metrics  [24]. 

2.1  Metrics  flow  in  the  CARDS  Program 

For  a  measurement  process  to  be  effective,  it  needs  to  be  an  integral  part  of  the  decision-making  process 
within  the  CARDS  Program.  Figure  2-ldepicts  the  interaction  and  flow  of  metrics  within  the  CARDS 
Program. 

On  the  left  hand  side  of  the  figure,  a  generalized  representation  of  CARDS  is  shown.  Starting  at  the  top 
left,  the  Customer  Goals  provide  context  for  the  development  of  the  CARDS  Program  Goals.  Customer 
Goals  are  stated  in  the  DoD  Software  Reuse  Vision  and  Strategy  [8],  by  ENS,  and  in  the  HAC  Operating 
Goals  and  Principles  [9].  These  goals  in  turn  direct  the  CARDS  team  in  the  development  and  execution 
of  the  activities  or  processes  that  constitute  the  existence  of  CARDS.  The  processes  ultimately  result  in 
CARDS  products  and  services  for  CARDS  users. 

The  metrics  flow  is  shown  on  the  right  hand  side  of  the  figure. 

1.  Metrics  definition  draws  on  the  CARDS  Program  Goals,  the  processes  implemented  by  the  CARDS 
team,  and  the  products  and  services  offered  to  CARDS  users. 

2.  The  meaics  definition  determines  the  collection,  analysis,  and  presentation  of  metrics  data. 

3.  Analysis  and  presentation  of  the  metrics  data  will  determine  what  findings  are  to  be  implemented. 
The  metrics  findings  and  the  changes  implemented  may  result  in  providing  input  for  changes  into 
Customer  Goals  as  they  apply  to  reuse  metrics,  CARDS  Program  Goals,  processes  and/or  products, 
and/or  the  metrics  definition. 
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2,2  CARDS  Prograni'Wide  Metrics  Goals 

For  a  measuremeat  process  to  be  successful,  it  must  be  guided  by  well-deficed  goals.  The  metrics  goals 
for  the  CARDS  Program  are  to: 

1.  Monitor  the  productivity  and/or  quality  of  a  process,  product,  and/or  service  by; 

a.  determining  trends  CARDS  is  interested  in; 

b.  identifying  what  CARDS  has  implemented  or  intends  to  implement  for  productivity  improvements 
(such  as  tool  usage  and  process  improvement); 

c.  validating  accepted  policies  and  procedures  or  indicate  areas  to  change;  and 

d.  determining  root  causes  for  defects. 

2.  Provide  visibility  into  projects  for  Ptogram  Management  by; 

a.  projecting  the  impact  of  resource  allocation  and  reallocation; 

b.  tracking  the  effect  of  events  on  processes  end  products;  and 

c.  maintaining  an  ongoing  profile/summary  of  CARDS  activities. 
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3.  Generate  preventable  quality  metrics  to  promote  CARDS  within  the  DoD  and  other  Government 
agencies,  as  well  as  within  industry. 

23  Goal/Question, Metric  Paradigm 

The  OQM  paradigm  provides  a  basis  for  defining  and  evaluating  a  set  of  operational  goals  using  mea- 
suremenL  This  paradigm  can  be  ^plied  to  each  goal,  process,  product,  and  service  offered  by  CARDS. 

The  OQM  paradigm  defined  by  Victor  Basil!  [24]  uses  a  top-down  approach  to  identitying  and  inter¬ 
preting  metrics.  To  implement  this  model  an  organiztuion  should: 

■  specify  the  goals  for  itself  and  its  projects: 

•  trace  goals  to  the  data  needed  to  quandfy  these  goals  operationally;  and 

•  provide  a  firamework  for  analyzing  the  data  in  the  context  of  the  goals. 

As  shown  in  Figure  2-2,  the  flow  from  the  goals  to  the  metrics  in  the  OQM  paradigm  can  ue  viewed  as 
a  directed  graph.  The  flow  is  from  the  goal  nodes,  to  the  question  nodes,  to  the  metric  nodes.  Each  goals 
generates  a  set  of  quantifiable  quesdons.  Each  quesdon,  in  turn,  generates  a  set  of  metrics  (mi.  is 
1.2...  Ji).  The  same  quesdons  can  be  used  to  define  multiple  goals,  and  several  related  metrics  may  be 
needed  to  provide  the  answer  to  one  question. 


Goal  1  Goal  2  Goal  3 


ml  m2  m3  m4  m2  m3  ml  m6  m7 


Example  of  Relationships  between  Goals,  Questions,  and  Metrics 
Figure  2-2.  Goals,  Questions,  and  Metrics 
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3  CARDS  Environment 

The  devdopraent  and  implementation  of  metrics  for  the  CARDS  Program  is  affected  by  the  need  for 
DoD-wide  metrics,  the  organizational  structure  of  the  CARDS  team,  the  CARDS  Program  with  its  pro¬ 
cesses,  products,  and  services,  and  the  establishment  of  franchises.  This  chapter  describes  the  challeng¬ 
es  faced  in  these  environments  and  assesses  the  readiness  of  the  CARDS  team  for  implementing  the 
measurement  process. 

3.1  DoD-Wide  Metrics 

The  DoD  Software  Reuse  Initiative  [9]  states  this  {Manciple  for  metrics: 

•  “Measure  Results  in  terms  of  customer  satisfaction.  Customer  success  produces  solid  referrals,  and 
customer  referrals  are  the  strongest  way  to  diffuse  technology.  Defining  success  in  customer  terms 
will  focus  and  integrate  the  activities  of  all  SRI  staff  members”. 

The  CARDS  tMm  should  design  surveys  on  user  satisfaction  that  provide  quantitative  feedback  on 
CARDS  products  and  services. 

The  DIS A/JIEO/CIM  Software  Reuse  Metrics  Plan  [7]  also  suggests  several  metrics  that  should  be  col¬ 
lected  by 

•  the  DoD  Program/Projea  Manager  on  the  products  and  services  provided  by  CARDS.  These  metrics 
should  be  fed  to  the  CARDS  Program;  and 

•  the  CARDS  Repository  Manager  on  the  CARDS  Library  System. 

Use  of  these  metrics  will  help  provide  more  focused  and  effective  services  to  usas  throughout  the  DoD. 
measure  the  payoff  from  the  reuse  initiative,  aid  developers  in  the  selection  of  reusable  components,  and 
provide  documented  evidence  for  successful,  frequent  reuse. 

3.2  Organizational  Structure 

The  CARDS  Program  is  composed  of  multiple  contractors  with  personnel  from  different  companies 
working  on  the  some  tasks.  Different  management  tools  and  techniques  are  used  to  collect  and  report 
on  data  within  each  company.  One  example  of  this  practice  is  the  timesheet  recording  mechanism.  At 
the  macro  level  each  company  charges  to  the  eleven  CARDS  task  areas,  while  at  the  micro  level  free¬ 
dom  is  given  for  a  finer  breakdown  in  recording  the  actual  work  performed  for  each  task,  the  layout  of 
the  timesheets,  the  collection  schedule,  etc.  For  a  measurement  process  to  be  successful,  consistent 
methods  must  be  devised  to  compensate  for  these  and  other  differences  in  data  cdlectioa  and  reporting. 

3  J  CARDS  Program 

3.3.1  CARDS  Processes 

The  processes  that  create,  refine,  and  transfer  CARDS  technology  are  being  developed  and  implement¬ 
ed  simultaneously  by  the  CARDS  team.  These  processes  are  at  various  stages  of  stability.  Some  pro¬ 
cesses  are  still  being  developed  (such  as  the  CARDS  organization  assessment  process),  while  others 
have  reached  stability  (such  as  the  Library  Administration  processes). 


Pages 


STARS-VC-B019A)04A)0 


31  Match  1994 


Metrics  will  be  most  effective  once  processes  have  become  stable.  Metrics  collected  in  an  unstructured 
environment  do  not  provide  data  that  can  be  compared  to  either  a  standard  or  to  another  process  in  an¬ 
other  environment  Too  many  variables  affect  the  data  to  make  evaluation  useful.  Table  1  describes 
when  a  process  is  deemed  stable  enough  for  metrics  definition  and  collection  to  be  effective. 


Table  1:  Process  Metric  Qualification 


Candklute  for  metirie  coUeefion..... 

now  being  defined 

immature 

not  documented 

premature 

documented,  but  changing 

qualifies 

stable,  rq)eatable,  used,  and  documented 

qualifies 

Processes  that  are  being  defined  or  that  are  undocumented  (and  thus  diffladt  to  rQ)eat)  are  not  qualified 
for  metrics  definitioa  However,  changing,  documented  processes  are  qualified  because  a  baseiine  mea¬ 
surement  can  be  taken  to  esublish  a  starting  point  for  improvement  The  improvements  suggested  will 
themselves  result  in  changes  to  process  steps.  These  changes  must  be  measured  to  ensure  that  any 
change  which  is  implemented  produces  the  desired  improvement,  and  to  measure  the  possible  impact 
of  the  change  on  other  steps  of  the  process.  Stable,  rq;)eatable  processes  are  also  qualified  for  metrics 
collection  which  will  help  ensure  that  the  processes  remain  stable  and  efficient. 

The  metrics  challenge  for  the  CARDS  team  will  be  to  evolve  the  measurement  process  as  CARDS  pro¬ 
cesses  mature.  One-time  activities  such  as  the  initial  learning  curve,  changing  processes  due  to  a  change 
in  direction,  or  experimenting  with  process  stqjs  should  be  eliminated  from  metrics  collectioa  They 
should  be  separated  from  repeat  activities,  such  as  qualifying  a  component,  installing  a  component  in 
the  CARDS  Library  System,  or  providing  training  to  a  prospective  franchise,  which  will  be  measured. 

3.3,2  CARDS  Products 

Products  provided  by  the  CARDS  Program  are  the  CARDS  Library  System  and  its  contents.The 
CARDS  Library  System  currently  consists  of  three  libraries:  the  Command  Center  Library  (CCL),  the 
PRISM  Distribution  Library  (PDL).  and  the  CARDS  Documentation  Library  (CDL). 

3.3.2.1  The  CCL 

The  CARDS  CCL  consists  of  reusable  components  at  multiple  levels  of  abstraction,  including  architec¬ 
tures,  requirements,  subsystems,  and  individual  components.  Given  the  model-based  approach,  the 
CCL  places  as  much  emphasis  on  the  complex  relationships  among  components  as  it  does  on  the  quality 
of  the  components  themselves.  One  aspect  of  the  CARDS  CCL  that  can  be  used  for  metrics  develop¬ 
ment  is  the  domain-specific  software  architecture,  which  serves  as  a  basis  for  "qualifying”  reusable 
components  for  the  library. 

Metrics  collected  on  CCL  components  should  take  into  account  the  component’s  various  implementa¬ 
tions.  (Qualification  criteria  to  determine  if  a  component  is  "fit  for  use”  should  be  established  for  each 
domain.  Certification  criteria  to  recommend  "best  of  class”  should  be  established  for  selected  compo- 
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nents.  Quality  factors  (such  as  Reliability.  Maintainability,  Usability,  Portability,  Security,  and  Perfor¬ 
mance),  their  priority,  and  their  respective  target  values  must  be  established  so  components  can  be 
evaluated  and  compared  objectively.  The  metrics  challenge  for  the  CARDS  team  will  be  to  define  and 
collect  metrics  during  and  after  the  development/evaluation  of  these  components  to  determine  how 
closely  a  particular  component  meets  its  expected  target.  Metrics  should  also  be  collected  firom  CCL 
users  as  individual  components  are  extracted  and  used,  to  Identify  user  preferences,  needs,  and  wants. 
This  data  can  be  used  to  tailor  the  contents  of  the  CCL.  Metrics  collected  on  the  “infrastructure**  of  the 
CCL  (the  semantic  network  and  the  RLF  browser)  will  be  useful  when  planning  for  improvements  to 
the  user  interface  and  the  speed  of  tool  response. 

3.3.2,2  ThePDLandCDL 

The  PDL  contains  PRISM  documents  such  as  quarterly  demonstration  rq)orts  and  software  develop¬ 
ment  plans  and  will  shortly  also  contain  wrappers  (inteiface  software).  The  CDL  will  contain  all 
CARDS  reusable  documentation  components  such  as  the  Library  Development  Handbook  and  the 
LOPP.  QualiQr  factors  for  these  products  should  be  established  so  measurement  can  help  to  determine 
their  Usability  and  Reliability. 

3.3J  CARDS  Services 

CARDS  services  exist  within  two  categories:  services  targeted  at  franchises  and  services  targeted  a  spe¬ 
cific  user,  as  well  as  services  provided  to  “spread  the  word**  about  reuse  technology  gained  by  C]ARDS. 
Examples  of  services  provided  to  franchises  and  users  are  organization  analysis.  Hotline  support,  and 
training  seminars.  Examples  of  services  provided  to  “spread  the  word*'  are  writing,  submitting,  and  pre¬ 
senting  pspers,  working  with  universities,  doing  demos,  and  attending  trade  shows. 

The  metrics  challenge  for  the  CARDS  team  will  be  to  construct  user  surveys  that  provide  quantitative 
feedback  on  CARDS  products  and  services.  These  surveys  allow  the  team  to  compare  user  expectations 
with  user  satisfaction  to  gauge  the  overall  effectiveness  of  the  services  provided  by  CARDS. 

Services  provided  to  ‘‘spread  the  word”  will  help  generate  interest  in  reuse  technology  and  thus  result 
(hopeftiUy)  in  more  franchises,  users,  and  students  who  are  familiar  with  reuse  technology.  Many  of 
these  benefits  are  long  term,  and  some  are  impossible  to  ascertain  (what  metric  value  to  attribute  to 
“word-of-mouth”  advertising  resulting  in  a  new  franchise  for  example.)  Due  to  the  expected  difficulty 
of  collecting  quant^able  benefits  on  these  activities,  it  is  recommended  not  to  measure  these  latter  ser¬ 
vices. 


3.4  Franchise  Implementation 

To  fulM  the  mission  of  technology  transfer,  the  CARDS  Program  is  establishing  franchises  with  fed¬ 
eral  agencies  to  fadlitate  the  Infusion  of  technology  developed  by  the  CARDS  team  and  other  reuse  re¬ 
searchers.  The  CARDS  team  is  defining  concepts,  sfrategies,  and  plans  (such  as  the  reuse  adoption 
handbook  and/or  the  franchise  plan  [11])  essential  to  supporting  reuse  via  franchising. 

Franchises  will  provide  CARDS  with  knowledge  about  their  reuse  experience.  This  knowledge  can  be 
provided  into  CARDS  processes,  an/or  additional  domain  expertise  in  the  form  of  new  components.  The 
metrics  challenge  for  the  C.\RDS  team  will  be  to  deflne  and  collect  metrics  from  each  franchise  to  de- 
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terciine  the  benefit  of  reuse  adoption.  The  team  should  also  measure  the  improvements  resulting  from 
process  feedback,  measure  the  cost  to  CARDS  associated  with  defining  the  concqits,  strategies,  and 
plans,  and  provide  a  method  to  CARDS  Program  Management  that  helps  allocate  the  cost  metrics  to 
each  franchise. 

Franchise  organizations  will  also  look  to  CARDS  for  guidance  on  defining  metrics  for  their  own  prod¬ 
ucts,  processes,  and  services  to  determine  their  own  cost/benefit  ratio  for  implementing  a  reuse  technol¬ 
ogy.  Metrics  should  be  defined  that  hdp  prospective  firanchises  determine  the  start-up  costs  of  building 
a  reuse  library. 

3.5  Assessing  the  CARDS  Measurement  Process 

To  assess  the  readiness  of  the  CARDS  team  fm*  a  measurement  process,  the  framework  to  evaluate  a 
measurement  process  proposed  by  Jeffery  &,  Berry  [10]  can  be  used.  The  framework  evaluates  the  con¬ 
text  in  which  a  measurement  process  is  developed  and  operated,  the  process  used  to  develop,  imple¬ 
ment.  and  maintain  the  measurement  process,  and  the  product  produced  by  the  measurement  process, 
such  as  the  rqxuts  produced.  This  framework  could  be  used  repeatedly  to  determine  quantitatively  how 
the  use  of  metrics  at  CARDS  is  improving.  The  evaluation  questions  and  comments  are  shown  in  tables 
1, 2,  and  3  in  Appendix  A. 

Preliminary  investigations  using  Jeffery's  framework  produced  these  observations; 

•  The  environment  in  which  the  CARDS  measurement  process  is  being  developed  and  operated  is  con¬ 

stantly  changing.  The  CARDS  team  is  working  to  establish  stable  goals  and  objectives,  and  to  doc¬ 
ument  processes.  Some  metrics  are  currently  being  collected  in  the  Library  Operations  project  area. 
As  CARDS  moves  towards  "proof-of-conc(q)t”  other  project  areas  will  also  be  stabilizing  their  pro¬ 
cesses.  The  metrics  analysts  must  be  involved  with  the  process  definition  to  ensure  that  process 
stqps  do  not  hinder  the  metrics  collection. 

•  The  metrics  themselves  do  not  currently  provioe  clear  benefits  to  management.  Metrics  collected  in 

the  Library  Operations  project  area  must  be  reevaluated  to  determine  why  specific  metrics  are  being 
collected.  The  next  chapter  outlines  a  set  of  matrices  that  help  trace  goals  to  metrics.  Lessons 
learned  during  the  definition  and  collection  of  metrics  for  Library  Operations  should  be  applied  to 
the  developmem  and  implementation  of  metrics  in  other  project  areas. 

•  Automation  of  the  methods  used  to  define,  analyze,  report,  and  maintain  CARDS  metrics  is  needed. 

Manual  data  collection  and  evaluation  is  labor  intensive,  time  consuming,  and  error  prone.  Tools  to 
automatically  collect  product  data  and  generate  a  metrics  database  are  needed  for  the  long  term  suc¬ 
cess  of  the  measurement  process.  While  Library  Operation  has  developed  scripts  to  help  collect 
some  of  the  data,  a  concentrated  effort  is  needed  to  evaluate  and  recommend  a  tool  and  database  for 
the  CARDS  measurement  process. 
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4  The  Measurement  Process 

The  measurement  process  is  a  dynamic  method  of  measuring,  assessing,  and  adjusting  products,  pro¬ 
cesses,  and  services  using  objective  data.  Metrics  are  collected  on  a  known  or  anticipated  issue,  concern 
or  question.  Metrics  are  analyzed  with  respect  to  the  characteristics  of  the  product,  process,  or  service 
and  are  used  in  turn  to  assess  their  respective  progress,  quality,  and  performance  throughout  develop¬ 
ment. 

This  chapter  outlines  a  six-st^  process  to  identify  CARDS  Program  goals,  processes,  associated  metric 
questions,  and  metrics;  and  to  collect,  analyze,  and  act  on  the  results  of  the  metrics.  This  chapter  also 
describes  a  set  of  matrices  within  the  measurement  process. 

4.1  Process  Overview 

The  measurement  process  is  dependent  on  CARDS  goals,  processes,  products,  and  services  that  are  to 
be  measured.  Improving  processes,  products  and/or  services  requires  continual  evaluation  of  past  expe¬ 
riences,  repeated  evaluation,  and  refinement  of  the  processes  used  to  produce  the  products  and/or  ser¬ 
vices. 

The  six-step  process  for  defining  and  implementing  metrics  is  Illustrated  in  Figure  4-1.  This  measure¬ 
ment  process  is  based  on  industry  experience  and  has  been  tailored  to  the  CARDS  Program. 


Metrics  Definitioa  Phase: 

Step  1:  Define  and  prioritize  program  and  project  goals 
Step  2:  Define,  refine,  and  document  processes 

Step  3:  Define  and  refine  metrics  questions  and  metrics 

Metrics  Implementation  Phase: 

Step  4:  Collect  the  metrics 

Step  5:  Analyze  the  metrics  data 

Step  6:  Act  on  results  of  the  analysis 


Figure  4-1.  The  Measurement  Process 

The  measuiement  process  is  divided  into  the  Metrics  Definition  Phase  (defining  goals,  processes,  and 
metrics)  and  the  Metrics  Implementation  Phase  (collecting,  analyzing,  and  acting  on  metrics  results). 
The  Mett  les  Definitioa  Phase  is  detailed  in  chapter  five.  The  Metrics  Implementation  Phase  is  detailed 
in  chapter  six.  The  arrows  In  Figure  4-1  show  bow  the  measurement  proce,ss  is  cyclical.  Acting  on  the 
results  of  the  analysis  in  Step  six  may  mean  refining  definitions  from  Steps  one,  two  and/or  three. 
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4.2  The  Matrix  Framework 

A  set  of  goal,  process,  and  metric  matrices  were  constructed  as  a  framework  for  documentiiig  and  di< 
recting  the  Metrics  Definition  Phase.  These  matrices  are  illustrated  in  Figure  4-2. 

4.2.1  The  Goal  Matrices 

Goals  provide  the  foundation  for  the  measurement  process  and  are  critical  to  its  success.  Inconectly 
identiiying  the  CARDS  Program  and/or  subsequent  project  goals  can  produce  disastrous  effects  on 
overall  Program  success. 

Each  goal  is  stated  from  a  particular  viewpoint  The  viewpoint  is  a  general  term  used  to  represent  the 
Interests  of  individuals  involved  with  the  CARDS  Library  System  (l.e.,  CARDS  team  members.  Project 
Leads,  DoD  customer,  Program  Manager,  System  Architect).  The  same  metric  could  be  interpreted  dif¬ 
ferently,  depending  on  whose  viewpoint  is  being  used.  For  example.  Library  Operations  may  need  to 
know  on  a  dally  basis  bow  the  hardware  is  performing  to  make  adjustments,  while  management  may 
only  need  to  know  on  a  monthly  basis  whether  the  hardware  continues  to  be  adequate.  The  metrics  that 
will  be  collected  should  be  designed  to  answer  concerns  from  each  of  these  viewpoints. 

Step  one  of  the  six-step  process  defines  the  goal  matrices.  Each  goal  matrix  shows  the  relationship  be¬ 
tween  two  sets  of  goals  (Customer  Goals  and  C^ARDS  Program  Goals,  CARDS  Program  Goals  and 
Project  Goals,  and  Project  Goals  and  Usa  Goals).  The  matrix  Customer  Goala/CARDS  Goals  lists  all 
DoD  Reuse  Directives  on  the  vertical  axis  and  all  CARDS  Program  Goals  on  the  horizontal  axis.  The 
intersection  of  each  DoD  Directive  and  CARDS  Program  Goal  shows  what  CARDS  Program  Goal  sup¬ 
ports  which  DoD  Direcdve.  More  than  one  CARDS  Program  Goal  may  map  to  a  DoD  Directive  and 
vice  versa.  For  the  purpose  of  metrics  definition  only  those  supporting  goals  that  map  to  an  individual 
goal  are  carried  over  to  the  next  matrix. 

CARDS  Program  Goals  are  carried  over  to  the  CARDS  Goals/Project  Goals  matrix  and  are  listed  on  the 
X-axis.  (Note  that  this  may  rqrresent  a  subset  of  the  originally  defined  goals.)  The  Intersection  of  the 
program  and  project  goals  shows  what  project  goals  support  which  CARDS  Program  Goals.  More  than 
one  project  goal  may  map  to  a  CARDS  Program  Goal  and  vice  versa. 

Project  Goals  are  then  carried  forward  to  the  Project  Goals/User  Goals  matrix,  which  shows  what  User 
Goal  mq>s  to  a  particular  Project  Goal.  The  intersection  of  Project  Goal/User  Goal  lists  the  products  and 
services  that  are  provided.  It  is  these  products  and  services  that  are  carried  forward  to  the  metrics  ma¬ 
trices. 

The  content  of  the  goal  matrices  is  assumed  to  be  somewhat  stable.  Program  and  project  goals,  once 
defined,  are  not  expected  to  change  drasdcully  during  the  year.  The  effort  to  define  these  three  matrices 
is  expected  to  be  a  one-time  effort  with  a  yearly  review  for  validation. 
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Figure  4>^2.  The  Metrics  Derinition  Framework 
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4.2.2  The  Process  Matrices 

Measuring  process  steps  is  a  very  important  part  of  the  CARDS  measurement  process^.  Improving  a 
process  for  the  purpose  of  defect  reduction  or  to  eliminate  non*value-added  steps  will  directly  result  in 
product  and  service  upgrades  as  well  as  cost  and  schedule  improvements.  TQM  activities  are  focused 
on  process  improvements  where  the  owners  (or  users)  of  a  process  are  responsible  for  improving  it.  For 
example,  domain  engineers  own  the  domain  en^neering  process  and  are  responsible  for  continually  im¬ 
proving  it  based  on  metric  findings.  The  same  focus  on  the  process  is  noticeable  in  SEI*s  Capability 
Maturity  Model  [3]  when  the  primary  objective  is  m  adiieve  a  controlled  and  measured  process  as  the 
foundation  for  continuous  improvement. 

The  processes  associated  with  defining,  managing,  implementing,  and  achieving  each  goal  are  listed  in 
the  process  matrices.  The  goals  from  the  goal  matrices  ate  carried  forward  to  the  process  matrices  which 
shows  the  relationship  between  processes  and  goals  (Customa  Goals  and  DoD  Processes.  CARDS  Pro¬ 
gram  Goals  and  CARDS  Program  Processes,  and  Frojea  Goals  and  Projecf  Processes),  For  instance, 
individual  Customer  Goals  am  listed  on  the  vertical  axis  and  supporting  DoD  Processes  are  listed  on  the 
horizontal  axis.  The  Intersection  (“X")  shows  which  process  is  used  to  implement  a  goal. 

Step  two  of  the  sbe-step  process  defines  the  processes.  DOD  processes  (such  as  contractual  processes, 
procurement  processes,  or  oversight  processes)  and  user  processes  an  outside  the  scope  of  the  Metrics 
Concqpt  Report  Only  CARDS  Program  processes  (such  as  processes  for  contract  management.  Tech¬ 
nical  Interchange  Meetings  (TIM),  CARDS  Configuration  Control  Board  (CCCB)  meetings,  informa¬ 
tion  dissemination,  system  growth  management)  and  project  processes  (such  as  the  domain  engineering 
process,  the  Library  release  process,  accounting  processes)  wUl  be  measured  for  process  feedback. 

The  content  of  these  matrices  is  assumed  to  be  somewhat  stable.  Program  and  project  processes,  once 
defined,  an  not  expected  to  change  drastically  during  the  year.  However,  process  changes  due  to  pro¬ 
cess  improvements  are  expected.  The  effort  to  define  these  three  matrices  is  expected  to  be  a  one-time 
effort  with  a  yearly  mview  for  validation. 

4.2  J  The  Metric  Matrices 

Two  more  matrices  are  needed  to  complete  the  Metrics  Definition  Framework;  the  Ooal/Product/Pro- 
cess/Service/Metric  (Question  matrix,  and  the  Metric  Question/Metric  matrix.  The  goals,  processes, 
products,  and  services  from  the  goal  and  process  matrices  an  carried  forward  to  the  metric  matrices. 
The  metric  matrices  show  the  relationship  between  the  goal,  product,  process,  and/or  services  and  the 
metric  questions  that  can  be  asked  on  them  and  the  metrics  that  must  be  collected  to  provide  the  answers. 


Aa  can  be  Inugined.  there  are  many  ways  to  achieve  a  result,  only  one  of  which  Is  the  desired  or  preferred  one.  For  eximple, 
If  the  desired  result  Is  “23  new  components  in  the  Library",  this  could  be  achieved  by  (s)  following  s  quality  selection  pro¬ 
cess,  (b)  taking  the  first  23  components  to  evaluate,  (c)  Uddng  the  23  easiest  components  to  evaluate,  (d)  dropping  in  any  23 
components,  and  so  on.  Either  proceu  gives  the  result  of  “23  new  components",  but  only  one  results  in  25  quality  compo¬ 
nents.  Unless  the  process  used  to  achieve  the  result  is  meuured  (l.e.,  made  visible),  management  will  never  know.  This  is 
why  it  is  so  important  to  focus  on  the  proceu.  It  also  illuitratu  why  it  is  fatal  to  i  quality  improvement  program  to  measure 
people  only  by  results. 
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The  metric  matrices  are  aeated  in  Step  three  of  the  six-sttp  process.  The  Ooal/Product/Process/Ser- 
vice/Metric  Questions  matrix  shows  what  metric  questions  are  asked.  The  Metric  Question/Metric  ma¬ 
trix  shows  what  metrics  are  used  to  answer  each  metric  question. 

These  two  matrices  are  seen  as  evolving  as  the  project  matures,  priorities  change,  and  experience  is 
gained  with  the  measurement  process.  They  need  to  be  updated  regularly. 

4  J.3.1  Defining  Metrics 

Metrics  can  be  broken  into  two  categories:  Result  metrics  and  Contributor  metrics.  Result  metrics  (“Ef¬ 
fects'*)  are  those  management-oriented  metrics  by  which  the  CARDS  Program  and/or  a  project  area  will 
be  judged.  Result  metrics  help  determine  if  a  goal  is  successftilly  achieved.  For  example,  if  the  goal  is 
to  “significantly  increase  the  contents  of  the  Library  as  measured  by  #  of  components  modelled'',  the 
result  metric  is  “#  of  components  installed''.  More  than  one  result  metric  may  exist  for  each  goal  or  ob¬ 
jective.  To  assist  with  manageability,  it  is  Important  to  identify  the  few  significant  metrics  for  each  goal. 

Contributor  metrics  are  the  technically  oriented  factors  (“Causes")  that  influence  judging.  If  the  result 
metric  is  “#  of  components  installed",  contributing  to  this  are  various  processes  (such  as  searching  for 
components  to  evaluate,  evaluating  components,  formatting  components  in  SGML)  that  must  be  under¬ 
taken  to  achieve  the  desired  result.  Each  contributing  process,  in  turn,  is  measured  by  Its  outcome  (con¬ 
tributor  metric,  such  as  “#  of  components  evaluated",  “#  of  evaluation  criteria  to  pass",  "days  to  format 
in  SGML").  There  can  be  many  contributing  processes  and  metrics  for  each  result  metric. 

Refinement  is  possible  where  the  contributor  metric  becomes  the  result  metric  and  smaller  processes 
contribute  to  it.  The  relationship  between  result  and  contributor  metrics  can  be  documented  in  a  Cause- 
and-Effect  diagram  (also  known  as  Ishlkawa  diagram,  or  Hshbone  diagram)  as  shown  in  Figure  4-3.The 
“Effects"  are  represented  on  the  hotixontal  axis  on  the  right  side  of  the  chart,  and  the  “Causes"  on  the 
diagonal  lines  are  listed  on  the  loft. 


Figure  4-3.  Cause  and  Effect  Diagram 


Page  13 


STABS-VC-B019/()04A)0 


31  March  1994 


Once  all  matrices  have  been  constructed  they  show  how  metric  questions  and  metrics  relate  to  the 
goal(s),  products,  processes,  and  services  they  support.  Questions  on  goals,  product  and  service  quality 
can  be  asked  at  any  level  (the  Result  Metrics).  Questions  related  to  the  processes  (i.e.,  the  contributing 
factors)  used  to  achieve  the  goals,  or  to  provide  these  products  and  services  can  be  asked  at  the  CARDS 
Program  and  project  levels.  Program  and  project  goals  flow  downward  to  support  the  metrics,  and  data 
flows  back  to  the  originating  question,  issue,  or  concern.  Maintaining  goals  using  this  framework  makes 
it  easy  to  add  new  metrics  and  metric  questions.  New  goals  can  be  added  with  immediate  knowledge  on 
the  impact  of  the  metrics  definition. 
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5  The  Metrics  Deflnition  Phase 

Basil! ’s  OQM  paradigm  will  be  used  during  the  Metrics  Definition  Phase.  DoD,  CARDS  Program,  and 
project  goals  and  their  associated  processes  are  defined  in  the  first  three  sets  of  matrices  and  carried  for¬ 
ward  to  the  last  two  matrices  where  metric  questions  and  metrics  are  defined.  A  Metrics  DeHnition  Da¬ 
tabase  to  hold  the  information  gathered  is  shown  in  chapter  seven. 

For  each  step  in  the  six-stq>  process,  the  Purpose  of  the  step  is  listed,  the  Participants  required  for  ex¬ 
ecuting  the  process  are  identified,  the  Input,  or  preparation,  necessary  to  execute  each  step’s  process  is 
discussed,  the  Process  to  implement  the  step  is  addressed,  and  the  Output  from  each  step  is  presenteu. 

5.1  Step  1:  Define  and  Prioritize  CARDS  Program  and  Project  Goals 

5.1.1  Purpose 

The  puipose  of  this  stq)  is  to  define  the  context  and  foundation  for  the  metrics  to  be  collected.  This  step 
is  critical  to  the  success  of  the  measurement  process.  The  DoD  vision,  CARDS  Program  Goals,  and  as¬ 
sociated  goals  for  domain  engineering,  library  operations,  franchise  implementation,  franchise  con¬ 
cepts,  training,  and  reuse  coordination,  must  be  defined  and  prioritized.  Users,  customers,  staff,  and 
management  have  different  needs  and  viewpoints  which  must  be  identified  to  Isolate  relevant  experi¬ 
ences  with  the  processes  and  on  other  projects. 

5.U  Participants 

The  working  group  should  be  comprised  of  the  Program  Manager,  the  System  Architect,  Frojea  Leads, 
the  metrics  analysts,  and  other  CARDS  team  members  as  needed.  Participants  in  the  working  group 
should  be  knowledgeable  about  the  goals,  products,  and/or  services  to  be  measured. 

5.U  Input 

Input  into  Step  one  is  any  existing  documentation  on  current  goals,  products,  and  services. 

5.1.4  Process 

The  task  of  defining  the  goals  and  identifying  viewpoints  will  be  accomplished  throughout  a  series  of 
interviews  and  brainstorming  sessions  with  CARDS  team  members.  Several  substeps  are  required  to  ex¬ 
tract  and/or  define  the  goals.  Separate  sessions  will  be  held  for  each  of  these  focus  areas: 

•  extract  the  Customer  Goals; 

•  define  the  CARDS  Program  Goals; 

•  define  the  Project  Goals.  (Note;  When  desirable  these  goals  can  be  refined  to  task  level  goals  before 
metric  questions  and  metrics  wUl  be  defined); 

•  define  the  services  and  products  provided  by  each  project  area;  and 

•  define  the  User  Goals.  The  start-up  nature  of  this  project  makes  it  difficult  to  determine  user  goals. 
A  suggestion  is  to  derive  user  goals  from  the  products  and  services  provided  by  each  project  area. 
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The  objectives  of  these  brainstorming  sessions  are  to; 

•  document  Customer  Goals  as  they  apply  to  the  CARDS  Program; 

•  define  and  prioritiKe  CARDS  Program  and  project  area  goals; 

•  define  the  services  and  products  provided  by  the  project;  and 

•  identify  user  goals. 

Transition  Technical  Interchange  Meetings  (HMs)  will  follow  each  brainstorming  session  to; 

•  validate  the  goals; 

•  update  the  next  project  team  that  will  work  on  the  next  focus  area  on  work  already  done;  and 

•  educate  CARDS  ttm  members  who  are  not  directly  involved  on  progress  made. 

Several  iterations  of  interviews,  brainstorming  sessions,  and  TIMs  may  be  necessary  to  completely  de¬ 
fine  each  matrix. 

5.1^  Output 

Goals  will  be  documented  in  three  matrices: 

•  Customer  Ooals/CARDS  Goals:  what  CARDS  Program  Goals  support  the  Customer  Goals; 

•  CARDS  Goals/Project  Goals;  what  Project  Goals  support  the  CARDS  Program  Goals;  and 

•  Project  Goals/User  Goals:  what  User  Goals  are  supported  by  what  Project  Goals,  Listed  at  the  inter¬ 
section  are  the  products/services  provided  by  the  project  area. 

The  matrix  Customer  Goals/CARDS  Goals  will  list  all  program  goals  for  the  CARDS  Prograno.  There 
may  not  necessarily  be  a  one-to-one  correlation  between  DoD  and  CARDS  Goals,  or  CARDS  Goals 
and  project  goals.  To  keep  the  process  manageable,  the  CARDS  Ooals/i  reject  Goals  matrix  may  actu¬ 
ally  be  a  set  of  matrices »  one  for  each  of  the  CARDS  projea  aieas.  Subse  luent  matrices  will  separately 
address  metric  questions  and  metrics  for  each  CARDS  project  area.  Note  that  the  Metrics  definition  Da¬ 
tabase  detailed  in  chapta  seven  would  help  significantly  with  the  manageability  of  this  effort. 

5.2  Step  2:  Deflne  /  Refine  /  Document  the  Processes 

5.2.1  Purpose 

The  purpose  of  this  step  is  to  define  and  document  the  processes  implemented  in  the  CARDS  Program 
or  a  project  area.  If  the  processes  to  be  measured  are  already  documented  or  If  a  product  quality  is  being 
measured  this  step  may  be  omitted.  It  is  strongly  recommended  to  measure  processes,  because  only  a 
change  in  process  steps  will  result  in  a  change  in  product  and/or  service  quality. 

All  processes  should  be  documented  in  as  much  detail  as  possible,  whether  or  not  metrics  are  collected. 
Basic  process  documentation  is  necessary  for  education  or  technology  transfer.  The  level  of  detail  to  be 
documented  for  each  process  for  metrics  collection  will  depend  on  the  detailed  questions  that  may  be 
asked  about  the  process.  However,  documenting  a  process  is  a  mqjor  effort  and  must  not  be  underesti- 
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mated.  Initially,  metrics  may  rnly  be  collected  at  a  very  high  level  to  gain  baseline  information  thus  al- 
leviatiiig  the  time  consuming  details  of  the  documentation  process. 

5.22  Partidpants 

The  working  group  should  be  comprised  of  Project  Leads,  the  metrics  analysts,  and  other  appropriate 
CAROS  team  members.  Participants  in  the  working  group  should  be  knowledgeable  about  the  goals, 
processes,  products,  and/or  services  to  be  measured  and  be  aware  of  the  results  of  the  measurement  pro¬ 
cess  so  far.  Members  of  the  working  group  should  also  be  familiar  with  basic  process  documentation 
techniques. 

5.2J  Input 

Input  into  Step  two  is; 

•  the  last  two  matrices  defined  in  Step  one  (i.e.,  the  CARDS  Ooals/Project  Goals  and  Project  Ooals/Us- 
er  Goals); 

•  a  list  of  processes  selected  by  Program  Management  and  Project  Leads  that  are  to  be  measured;  and 

•  any  existing  documentation  on  those  processes. 

5.2.4  Process 

Members  of  the  working  group  will  initially  select  the  best  documentation  method  (such  as  SADT, 
IDEF,  or  flowcharting)  to  be  used  for  all  processes  to  be  defined.  Use  of  the  same  methodology  will 
promote  consistency  in  understanding  each  others  work  and  conveying  correct  information  among  the 
participants.  This  transfer  of  knowledge  becomes  critical  during  Step  four  (collecting  metrics)  of  the 
measurement  process. 

Several  working  sessions  will  be  required  to  document  and  review  the  processes.  The  objective  of  these 
working  sessions  is  to  document  the  processes. 

A  transition  TIM  will  be  held  following  this  definition  to  educate  the  CARDS  team  and  update  the 
project  twf"  that  will  work  on  the  next  stq).  FoUow-on  working  sessions  to  further  refine  the  processes 
may  be  needed. 

S.2JS  Output 

The  outcome  of  this  effort  is: 

•  two  matrices  (the  CARDS  Goals/CARDS  Process  and  Project  Ooals/Project  Process)  showing  which 
processes  are  used  to  define,  manage,  'mplemeut,  and  achieve  each  goal;  and 

•  the  documented  processes  themselves. 

Due  to  the  expected  volume  of  process  documentation,  an  inr^'-'idual  may  be  .i.  to  help  maintain 
this  documentation. 
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53  Step  3:  Define  /  Refine  Metric  Questions  and  Metrics 
53.1  Purpose 

The  purpose  of  this  step  is  to  define  the  metric  questions  and  their  metrics,  select  the  tools  and  tech¬ 
niques  to  be  used  to  collect  the  metrics,  set  the  standards  of  performance  against  which  actual  results 
will  be  compared,  and  define  the  reports  and  procedures  for  reporting  actual  results. 

533  Participants 

This  working  group  should  be  chaired  by  the  metrics  analysts  with  members  selected  by  the  correspond¬ 
ing  Project  Leads.  Participants  of  the  working  group  should  be  knowledgeable  about  the  goals,  process¬ 
es,  products,  and/or  services  to  be  measured  and  be  aware  of  the  results  produced  by  the  measurement 
process  thus  far. 

5.33  Input 
Input  into  Step  three  is: 

•  the  previously  defined  matrices  and  processes:  and 
■  existing  metric  q*jestions  and  metrics. 

.5.3.4  Process 

Several  sessions  will  be  required  to  define  metric  questions,  metrics,  collection  methods,  and  rqx>rts  for 
each  goal,  process,  ptodua,  <x  service.  The  objectives  of  the  working  group  are  to; 

•  develop  a  set  of  metrics  questions:  and 

•  develop  a  list  of  key  metrics  to  be  collected. 

For  each  metric  that  is  to  be  collected,  the  working  group  will  determine: 

•  assumptio.is,  constraints,  and  other  knowledge  about  the  metric: 

•  the  metric  collection  method  (user  feedback,  system  generated,  from  a  log  book): 

•  the  metric's  reliability,  correctness,  and  validity: 

•  who  wiU  be  responsible  for  collecting  the  metric): 

•  the  standard  against  which  the  metric  wUl  be  compared: 

•  how  often  the  metric  is  to  be  collected  and  presented;  and 

•  the  format  that  will  be  used  (i.e.,  report,  graph,  or  list)  to  collect  and  present  the  metric. 

A  transidoo  TIM  will  be  held  following  this  definidon  to  educate  the  CARDS  team  and  update  the 
project  team.  Fdlow-on  working  sessions  to  further  refins  ibe  metric  quesdons  and  metrics  may  be 
needed. 
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5.3^  Output 

The  outcome  of  this  effort  will  be  two  sets  of  work  products: 

(1)  two  more  matrices: 

•  Goal/  Product/  Frocess/Service//Metric  Questions:  which  metric  questions  must  be  answered 
for  which  goahlprocess/servicc/product;  and 

•  Metric  Questions/Metric:  what  metrics  support  what  metric  question(s). 

(2)  a  Metric  Worksheet  as  shown  in  Figure  5-1  that  shows  for  each  caetric: 

•  the  goal  the  metric  is  tied  to; 

•  the  metric  name  and  its  measuring  unit; 

•  the  data  source  of  the  metric; 

•  the  data  collection  method: 

•  the  collection  schedule: 

•  the  responsible  party  to  collect  the  metric: 

•  any  assumptions  and  constraints  known  about  the  metric: 

•  the  recommended  standard  for  this  metric, 

•  the  justiflcatiou  for  the  standard:  and 

•  any  Program  Management  comments. 
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Metric  Metric  Unit  Data  Source  Collection  Method  CoUecdon  Schedule  Resp.  Party 


Assumptions,  Constraints,  and  Other  Knowledge  about  Metric: 


Program  Management  Comments: 


Figure  5-1.  Metric  Worksheet 
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6  The  Metrics  Implementation  Phase 

Steps  four  through  six  make  up  the  Metrics  Implementation  Phase.  Automated  data  collection  reduces 
the  ovahead  on  the  CARDS  team  members  charged  with  collecting  the  data.  It  needs  to  be  determined 
what  data  is  already  collected  by  a ’tool”  currently  used  (si  ‘  as  OS  commands  executed,  terminal  ID, 
date  and  time,  amount  of  CPU  time,  etc.)  and  what  data  is  reworded  manually. 

A  sqiarate  effort,  prioritized  against  other  work,  needs  to  be  undertaken  to  determine  what  tools  will  be 
needed  to  automate  the  data  collection,  data  submission,  data  storage,  data  analysis,  and  report  ftmc- 
dons.  Tools  such  as  Amadeus  [  16]  to  collect  metrics  data,  and  statistical  analysis  and  presentation  pack* 
ages  will  bo  analyzed  to  assist  iu  the  automation  effort.  The  effort  required  to  automate  these  activities 
and  manage  the  database  needs  to  be  justified  by  the  effort  required  currently  to  manually  collect,  ana¬ 
lyze,  and  present  the  data. 

6.1  Step  4:  Collect  the  Metrics 

6.1.1  Purpose 

The  puipose  of  this  step  is  to  collect  metrics  to  uncover,  document,  compile,  and  rank  problems.  Metrics 
help  answer  questions  on  current  status,  shortcomings,  and  ftiture  plans^ 

6.U  Participants 

CARDS  project  team  members  executing  the  work  processes  in  their  day-to-day  work  will  collect  the 
metrics.  Team  members  should  be  aware  of  all  results  produced  thus  far  by  the  measurement  process, 

6.1J  Input 
Input  into  Step  four  is 

•  the  Metric  (^uestion/Metrlcs  matrix,  and 

*  the  Metric  Worksheets  developed  in  Step  three. 

If  tools  exist  to  automate  data  collection  they  must  be  implemented. 

6.1.4  Process 

CARDS  team  members  will  execute  the  work  process,  foUowmg  the  process  steps  documented  in  Step 
two.  Process  and  product  data  will  be  collected  as  part  of  the  day-to-day  work  activities.  Data  fiom  the 
user  should  also  be  collected.  The  frequency  of  data  collection,  the  collection  method,  and  the  respon¬ 
sible  party  to  collect  each  metric  are  indicated  on  the  Metric  Worksheets  for  each  metric.  Some  of  the 
data  collected  can  be  used  to  provide  real-time  feedback  to  the  project  organization  and  the  metrics  an¬ 
alysts,  for  immediate  process  control.  Care  must  be  taken  to  ensure  that  data  is  not  manipulated  and  that 


Note  that  metrics  will  show  the  symptoms  experienced  by  the  user  of  a  process  w  product/service, 
not  a  problem  itself.  For  example,  if  a  metric  shows  tea  defects  uncavered  during  testing,  the  problem 
is  not  that  there  are  ten  defects.  The  problem  is  most  likr'j  'u  the  process  used  to  produce  the  soft¬ 
ware. 
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consistency  is  maintained  across  the  many  sources  of  data.  Given  the  expense  of  coliecting  data  only 
data  with  a  defined  use  sbouid  be  collected. 

6.U  Output 

The  ouh^ut  of  this  effort  will  be  raw  metrics  data,  stored  on-line  or  on  data  sheets. 

6.2  Step  5:  Analyze  the  Metrics  Data 

6.2.1  Purpose 

The  purpose  of  this  step  is  to  identify  the  problems,  rank  them  In  some  agreed-upon  order,  identify  the 
root  cause(s)  of  the  problem(s),  and  suggest  possible  sOlutioos. 

6,2J2  Participants 

The  working  group  should  be  comprised  of  the  CARDS  Project  Leads,  the  metrics  analysts,  and  other 
CARDS  team  members.  Participants  of  the  working  group  should  be  knowledgeable  about  the  process¬ 
es,  products,  and/or  services  to  be  analyaed. 

6.23  Input 

Input  into  Step  five  is: 

•  the  metrics  collected  in  the  previous  step: 

•  all  matrices  developed  in  the  Metrics  Definition  Phase;  and 

•  the  relevant  Metric  Worksheets. 

6.2.4  Process 

Various  Quality  Control  (QC)  techniques  can  be  used  to  help  identiiy  and  rank  the  mt^or  problems; 
Pareto  charts,  cause  and  effea  diagrams,  scatter  plots,  check  sheets,  pie  charts,  histograms,  graphs,  an¬ 
d/or  control  charts.  The  team  will  decide  which  technique  best  suits  its  needs.  Various  criteria,  such  as 
severity,  frequency,  and/or  cost  can  be  used  to  determine  the  priority  with  which  problems  should  be 
addressed.  The  team  must  also  identify  the  root  causeCs)  of  the  problem(s)  by  Identiiyiog  all  process 
components  related  to  creating  the  problem^  This  is  where  analysis  of  the  metrics  within  the  context  in 
which  they  were  collected  will  yield  valuable  informatioa  for  process  improvements.  Related  program 
issues  and  risks,  corrective  action  plans,  standards,  tralning/skiUs,  and  procedures  could  all  be  related 
to  the  problem.  Finally,  the  working  group  must  identify  and  suggest  possible  solutions  and  the  best  way 
to  implement  change. 


Identifying  and  ranking  the  problem  ii  producllnervtce  related.  Identifying  the  root  cause  and  suggesting  a  pos¬ 
sible  loludofl  is  process  related. 
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6.2.5  Output 

Output  from  this  stq>  will  be  the  various  QC  charts,  as  well  as  suggestions  for  process  improvements. 
Regular  Program  Management  Reviews  (PMRs)  for  CARDS  will  provide  visibility  for  this  effort. 

63  Step  6:  Act  on  Results 

6.3.1  Purpose 

The  purpose  of  this  step  is  to  gain  management  approval  to  Implement  the  identified  changed  to  take 
possible  short-term  action,  and  to  implemem  long-term  solutions  in  process  improvements. 

6.3.2  Participants 

This  group  should  be  comprised  of  the  Program  Manager,  the  System  Architect,  and  Project  Leads. 
6,33  Input 

Input  into  Step  six  is  the  analysis  work  done  in  Stq;>  five. 

6.3.4  Process 

CARDS  Program  Management  must  make  it  a  priority  to  actively  address  issues  revealed  by  metrics. 
The  Program  Manager  will  approve  recommended  process  changes  betbre  they  are  Implemented.  Im¬ 
plementing  the  solution  can  be  broken  into  short  term  and  long  terms  activities.  Since  process  improve¬ 
ments  may  take  several  weeks  or  months  to  complete,  the  team  should  implement  any  action  items  that 
can  be  accomplished  in  the  short  lem  (such  as  inserting  a  warning  message,  installing  a  workaround). 
Any  process  changes  must  be  verified  to  eiBure  that  the  desired  result  is  indeed  achieved.  This  may 
mean  cycling  back  to  Step  four,  “Collect  metrics”  and  comparing  the  now  results  with  the  old  ones. 

6.3.5  Output 

Process  changes  must  be  documented  as  outlined  in  Step  two.  Reuse  experience  and  lessons  learned 
through  metrics  collection  will  be  added  to  each  CARDS  project  area.  Charts  depicting  product  growth 
over  time  and  associated  confidence  levels  will  be  maintained. 


The  ciiterle  used  to  approve  a  solution  must  be  stated  to  the  team  btfore  wodc  is  done  on  problem  analysis  and 
a  solution  is  suggested.  Disagreeing  with  a  solution  at  this  step  because  the  wrong  evaluation  criteria  were  used 
means  the  team  has  to  go  back  to  Step  Ave.  This  wastes  time.  To  say  nothing  of  the  frustration  to  the  team. 
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7  Added  Value  Through  Automation 

A  rdational  database  can  be  developed  to  hdp  the  metrics  definition  process.  An  application  with  views 
into  the  database  can  be  developed  that  steps  through  the  process  of  defining  the  goal,  process,  and  met¬ 
ric  matrices. 

The  design  of  the  Metrics  Definition  Database  is  depicted  in  Figure  7*1.  Goals,  processes,  services  and 
products,  as  wdl  as  the  associated  metric  questions  and  metric  definitions  are  stored  in  the  database. 
The  metrics  analyst  can  thus  easily  compare  new  items  to  those  already  defined  in  the  system.  Oianges 
to  the  definition  of  any  part  of  the  system  can  be  traced  to  all  affected  parts  through  links  defined  in  the 
relational  database. 

If  an  automated  metrics  collection  tool  such  as  Amadeus  is  used,  and  links  between  the  collection  tool 
and  the  metrics  definition  database  are  possible,  then  the  Metrics  Definition  Database  could  be  used  to 
drive  an  automated  collection  .A  metrics  data.  StatiKlcal  analysis  packages  to  help  with  the  data  evalu¬ 
ation,  and  presentation  packages  should  also  be  evaluated  for  automation.  This  will  be  fluther  addressed 
in  each  project  area's  Metrics  Plan. 

7.1  The  Metrics  Dennitfon  Database 

Goals  are  defined  in  “G"  tables.  The  Customer  Goals  are  stored  in  the  Glx  table,  the  CARDS  Program 
Goals  are  stored  in  the  G2x  uble,  and  the  DoD/Cards  Program  Goals  matrix  is  stored  as  the  Glx-02x 
table.  Each  record  in  the  01x-G2x  table  corresponds  to  one  element  in  the  matrix.  CARDS  Project 
Goals  are  stored  in  the  G3x  table,  and  the  relationship  between  the  Program  and  project  goals  is  stored 
in  the  Q2x-03x  table.  This  corresponds  to  the  second  matrix  “CARDS  Progran^ojea  Goals."  User 
Goals  are  stored  in  the  G4x  table  and  the  relationship  between  user  goals  and  project  goals  is  stored  in 
the  G3x-04x-S4x  table.  This  table  corresponds  to  the  third  matrix  “Project  GoalsAJser  Goals,"  the  in¬ 
tersection  of  which  lists  the  products/services  provided. 

Processes  are  defined  in  “P”  tables.  The  P2x  table  contains  the  processes  used  at  the  CARDS  Program 
level  and  the  G2x-P2x  table  corresponds  to  the  CARDS  Program  process  matrix  which  lists  the  process¬ 
es  used  to  support  the  CARDS  Program  goals.  The  P3x  table  contains  the  CARDS  Project  processes, 
and  the  03x-P3x  table  corresponds  to  the  CARDS  Project  process  matrix  which  lists  the  project  pro¬ 
cesses  needed  to  support  the  project  goals. 

Metric  questions  and  metrics  are  defined  in  the  “Q”  and  “M”  tables  respectively.  The  Metric  Question 
table  contains  the  questions  derived  from  the  goal,  process,  or  producn/service  statements.  The  Mx  table 
contains  the  metrics  definitions  and  the  source  of  each  metric,  the  frequency  of  collection,  the  respon¬ 
sible  party,  and  other  Infomution  from  the  Metric  Worksheet. 
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Figure  7-1.  Metrics  DeHnition  Database 

_____ 
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8  Conclusion 

It  is  tbe  overall  measurement  process  not  just  tbs  graphs  and  rqx>rt$  that  adds  value  tu  tha  data.  The  six- 
stq)  process  outlined  in  this  document  is  integral  to  the  C  AIIDS  Program  and  shows  how  the  knowledge 
about  the  data,  the  goals  the  data  is  tied  to,  and  the  viewpoints  affected  are  all  necessary  to  understand 
what  the  data  means  and  how  it  is  to  be  used.  As  the  measurement  process  is  being  implemented,  lessons 
learned  from  it  will  be  used  to  further  refine  it  This  requires  active  management  commitment  and  sup¬ 
port.  To  successfully  implement  the  measurement  process,  it  is  necessary  to: 

*  know  what  the  goals  are.  They  drive  the  measurement  process  and  the  CARDS  Program; 

*  identify  the  key  metrics  by  which  the  CARDS  Program  will  be  evaluated: 

*  apply  the  discipline  to  follow  documented  proce.ss  stq>s.  If  steps  are  skipped,  the  process  becomes 
unknown  and  its  data  will  bo  useless; 

*  collect  metrics  on  CARDS  processes  to  help  mate  these  processes  visible  and  thus  become  a  candi¬ 
date  for  process  improvement.  Process  Improvements  will  result  in  product  and  service  improve¬ 
ments,  as  well  as  cost  and  schedule  improvements:  and 

*  automate  as  much  as  possible  the  definition,  collection,  analysis  and  rqmrting  of  the  metrics. 

As  CARDS  processes  continue  to  mature  the  challenges  faced  by  the  CARDS  team  Include  evolving 
the  measurement  process  and  devising  methods  that  compensate  for  differences  in  data  collection  and 
reporting  among  the  eleven  companies  comprising  the  CARDS  team.  The  CARDS  team  is  also  faced 
with  conceiving  new  and  creative  ways  of  measuring  tbe  level  of  user  satisfaction  with  the  CARDS  Li¬ 
brary  System. 

The  effort  needed  to  define  the  goals  and  processes  is  significant  and  can  easily  span  a  year  or  more.  In 
the  meantime,  baseline  measures  can  be  taken  in  some  project  areas  (such  as  in  Library  Operations)  to 
gain  an  understanding  of  where  process  improvements  are  possible.  A  phased  implementation  to  gain 
experience  with  the  measurement  process  is  also  recommended. 

8.1  Next  Steps 

The  first  CARDS  Project  Area  to  benefit  from  the  six-step  measurement  process  wUl  be  the  CARDS 
Library  System.  Metrics  are  already  collected  in  isolated  areas,  such  as  in  Library  Administration.  The 
six-step  process  will  take  advantage  of  already  existing  metrics  and  build  from  there.  Given  the  time 
schedule  (Phase  m  will  be  completed  the  etui  of  March  1994),  only  a  subset  of  metrics  will  be  collected. 
Due  to  the  expected  effort  and  time  required  to  define  the  goal  and  process  matrices,  tbe  initial  effort 
will  focus  on  key  issues  that  have  already  been  identified,  such  as  the  Lllvary  release  process.  SCIR/STR 
metrics,  graphical  presentation  of  metrics,  and  automation  needs.  Parallel  to  this  activity,  high-level 
goals  will  be  extracted  and  defined.  Matrices  will  thus  be  filled  in  from  tbe  top  and  the  bottom.  Once 
work  has  been  documented  in  this  fashion,  additional  goals  and  metrics  can  be  defined. 
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Appendix  A  A  Framework  to  evaluate  a  Metric 

Program’s  Success 

The  table  bdow  lists  the  assessment  criteria  that  can  be  used  to  evaluate  the  metrics  program  at  CARDS. 
These  questions  can  be  used  to  periodically  evalutde  the  CARDS  Program,  noting  improvements. 
These  criteria  are  taken  directly  fiom  Jeffiey  &  Berry  [10].  A  scoring  mechanism  of: 

0«  does  not  meet  any  of  the  requirement 

!■  meets  some  of  the  requirement 

2m  meets  most  of  the  requirement,  and 

3*  ftiUy  meets  the  requirement 

could  be  applied,  with  equal  weight  given  to  each  quesdoa  This  assessment  could  be  performed  before 
establishing  metrics  in  a  particular  CARDS  project  area,  and  then  periodically  to  help  determine  how 
successful  the  metrics  program  at  CARDS  is. 


Ihble  3:  **Context”  is  the  environment  in  which  the  CARDS  Metrics  Program  is  being 

developed  and  operated. 


QiutSoH 

Comments 

Are  the  goals  of  the  CARDS  measurement 
program  congruent  with  the  goals  of  the 
CARDS  Program? 

CARDS  mettwement  goals  tx  an  integral  part  of 
dm  CARDS  Program.  Chapter  4  shows  how 
nmtiiei  are  tied  to  CARDS  Program  sod  prqject 
goals. 

Are  the  objectives  and  goals  for  the 

CARDS  measurement  program  clearly 
stated? 

Ooah  and  objeedvea  for  the  CARDS  measurement 
program  are  delhied  in  chapter  2  of  dds  document 

Does  the  CARDS  measurement  program 
have  a  realistic  assessment  of  pay-back 
period? 

The  length  of  the  payback  period  will  depend  on 
the  nun^  and  dep^  of  process  improvements 
Identified  and  implemented.  Metrics  to  help  eati> 
mate  the  start-up  cost  of  building  a  reuse  Library 
need  to  be  established  for  franchises. 

Does  the  measured  staff  participate  in  the 
development  of  the  measures? 

Staff  will  not  be  meuured.  Stsff,  however,  should 
participate  in  the  motrlcs  definition  and  implemen- 
tadoo  process  and  help  widi  the  analysis  process. 

Has  a  quality  environment  been  estab¬ 
lished? 

Quality  is  stressed  dirou^iout  the  CARDS  Pro- 
gram. 

Are  the  processes  stable? 

Many  processes  such  as  the  domain  engineering 
process  are  sdll  being  defined.  Processes  must  be 
repeatable  for  metrics  coUecdoo  and  analysis  to  be 
effective. 

Can  the  required  granularity  of  data  to  be 
collected  be  determined  and  is  the  data 
available? 

Data  granularity  is  determined  by  the  metric  ques- 
dons  asked  on  a  particular  goal,  process,  product, 
or  service.  Data  is  svailable  today  in  library  Oper- 
adons.  Data  on  other  project  areas  wUl  become 
available  as  the  measureoient  process  matures. 
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Table  3:  ‘‘Context”  is  the  environment  in  which  the  CARDS  Metrics  Program  is  being 

developed  and  operated. 


Qutsdott 

Comments 

Is  the  measurement  program  tailored  to 
the  needs  of  the  organization? 

The  metrics  deflnitioa  framework  is  designed  to  be 
tsilorable. 

Is  senior  management  commitment  avail¬ 
able? 

Senior  maiiagement  has  stated  its  commitment 
rqxatedly. 

Table  4:  “Process”  is  the  method  used  to  develop,  implement,  and  maintain  a  CARDS 

Metrics  Program. 


Question 

Comments 

1  Process  Motivation  and  Objectives  | 

Is  the  program  promoted  through  the  pub¬ 
lication  of  success  stories  and  encourag¬ 
ing  exchange  of  ideas? 

CARDS  does  many  publlcatioos  to  exchange  ideas 
on  muse.  Metrics  success  stories  are  not  yet  pub* 
lisbed. 

Is  a  firm  implementation  plan  published? 

This  document  provides  the  foundation  for  the 
CARDS  Metrics  Plan. 

Does  the  program  assess  individuals? 

Metrics  should  never  be  used  to  assess  individuals. 

1  Process  Responsibility  and  Metrics  Ibam  | 

Is  the  metrics  team  independent  of  the 
software  developers? 

An  independent  metrics  team  can  concentrate  on 
metrics  definition  and  help  with  the  analysis  in  a 
less  biased  manner.  The  metrics  analysts  an  inde* 
pendent  of  the  development  team. 

Are  clear  responsibilities  assigned? 

Project  Leads  are  responsible  for  assigning  project 
responsibilities.  Key  individuals  an  assign^  tte 
nspoosibllity  for  the  metrics  concept  hnplementa- 
don. 

Is  the  initial  collection  of  metrics  sold  to 
the  data  collectors? 

Staff  collecting  the  data  must  know  why  the  data  is 
collected,  and  what  the  data  is  used  for.  Thin  will 
help  establish  a  posidve  ctdtude  towards  data  col- 
lecdoo.  Staff  will  participate  in  the  metrics  definl- 
don  as  outlined  in  chapter  4. 

1  Process  Data  Collection  | 

Are  the  important  initial  metrics  defined? 

Some  metrics  are  defined  through  internal  agree¬ 
ments  with  government  program  representadves. 

Are  tools  for  automatic  data  collection 
and  analysis  developed? 

Yes.  Locally  developed  scripts  are  used  in  Library 
Operadoos.  More  needs  to  te  done  such  as  the 
evaluadon  of  Amadeus  and  similar  tools  that  facil¬ 
itate  coUecdon  and  on-lioe  presentadon. 

Is  a  metrics  database  created? 

Not  yet.  A  Metrics  Detlnidon  Database  is  proposed 
in  this  document.  A  database  containing  metrics 
data  should  also  be  developed. 
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Table  4:  'Trotess'’  is  the  method  used  to  develop,  implement,  and  maintain  a  CARDS 

Metrics  Program. 


Qutstton 

Comments 

Is  there  a  mechanic  for  changing  the 
measurement  system  in  an  orderly  way? 

The  Metrics  Concept  Report  proposes  the  use  of 
Bssili’s  GQM  paradigm  to  define  metric  questions 
and  metrics.  The  Metrics  Plan  will  outline  the  pro< 
cess  for  controlled  changes  to  software  and  man* 
agement  piocesaes  based  on  metrics  findings.  The 
proposed  database  will  help  identify  all  go^, 
questioos,  and  metrics  afiGected  by  changes  to  any 
ot  these  components. 

Is  measurement  integrated  into  individual 
CARDS  processes? 

Ib  provide  useful  information  on  process  improve* 
meats,  metrics  must  be  part  of  a  process,  not  sepa* 
rale  from  it  Metrics  are  integrated  into  library 
Operadons  processes  and  must  be  integrated  into 
the  other  project  area’s  processes. 

Are  capabilities  provided  for  users  to 
explain  events  and  phenomena  associated 
with  the  project? 

Measurement  without  context  can  be  misleading. 
Metrics  must  be  evaluated  in  light  of  assumptioas, 
constraints,  and  other  knowledge  for  a  tree  picture 
to  emerge.  The  metric  worksheet  proposed  In 
Chapter  5  helps  maintain  pertinent  information  for 
each  metric. 

Is  the  data  collected  cleaned  Le.,  normal¬ 
ized,  and  used  promptly? 

Data  is  not  currently  normalized.  Metrics  collected 
in  the  Library  Operations  area  are  used  in  PMR 
presentatloos,  but  not  otherwise.  As  the  measure¬ 
ment  process  is  implemented,  data  will  be  used  to 
help  with  process  improvements.  Impact  assess¬ 
ments  and  recommendatioos  are  nee^  in  all 
project  areas. 

Do  the  CARDS  Program  Goals  determine 
the  measures? 

Chapter  2  of  this  document  outlines  the  Basili 

GQM  paradigm.  Chapter  4  shows  how  to  tie 

CARDS  Program  go^  to  metrics. 

1  Process  Training  and  Awareness  | 

Is  adequate  training  in  metrics  carried 
out? 

Training  in  the  purpose  and  use  of  metrics  is 
required  for  everyone  involved  with  the  CARDS 
Pr^am.  Management  must  make  data-based 
dedsions,  staff  must  understand  the  cuUectioo  and 
help  with  the  analysis,  and  Project  Leads  must 
change  process  steps  based  on  metrics  findings. 

The  measurement  plan  outlined  in  chapters  S  and  6 
shows  where  informal  TIMs  are  to  be  used  to  train 
anyone  involved  in  metrics. 

Does  everyone  know  what  is  being  mea¬ 
sured  and  why? 

As  metrics  are  defined  in  each  project  area,  every¬ 
one’s  help  will  be  needed  to  improve  the  processes 
for  better  products  and  services.  This  involvement 
will  provi^  individuals  with  insight  into  what  is 
being  measured  and  why. 
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HBble  5:  ‘Products”  are  the  measures  taken  on  CARDS  products,  services,  and 
processes,  the  reports  produced,  and  other  outputs  of  the  Metrics  Program. 


Qutxtbn 

Comments 

Are  the  measures  clear  and  of  obvious 
applicability? 

Bssili's  QQM  paradigm  ties  measures  to  dieir 
goals.  Knowing  what  the  question  is.  and  which 
goal  it  is  tied  to.  helps  to  dearly  define  the  metrics 
that  must  be  collect^. 

Do  the  measures  taken  provide  clear  bene¬ 
fits  to  the  management  process? 

Only  Library  Qperadoos  currently  collects  metrics. 
Metrics  must  be  defined  that  will  impact  the  man* 
agement  processes  in  all  CARDS  project  areas. 

Is  feedback  on  results  provided  to  those 
being  measured? 

Processes,  services,  and  products  are  measured, 
not  people.  Those  collecting  metrics  on  a  process, 
pro^t.  or  service  must  also  receive  the  results  of 
the  wyifttc-n  analysis  to  understand 
dedsions  and  add  analysis  details  (“context")  to 
the  data. 

Is  the  measurement  system  flexible  enough 
to  allow  for  the  addition  of  new  tech¬ 
niques? 

Yes. 

Are  measures  used  only  for  pre-defined 
objectives? 

Metrics  that  are  not  tied  to  goals  and  objectives  of 
the  CARDS  Program  and  processes  yidd  no  uae^ 
infonnation. 
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Appendix  B  Interoperability 

This  appendix  is  an  example  that  shows  how  information  gained  from  the  interoperability  project  is 
used  to  construct  a  set  of  five  matrices.  The  matrix  numbering  scheme  is  shown  in  Hgum  B-1. 
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Sub  Goal  1 

Sub  Goal  2 
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Sub  Goal  4 
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Figure  B-1  The  Matrix  Numbering  Scheme 

If  Sub  Goal  1  is  supported  by  Goal  1,  an  *'X"  will  be  placed  in  position  1.1.  If  Sub  Goal  1  is  also  sup¬ 
ported  by  Goal  3,  another  “X”  will  be  placed  in  position  3.1. 


Some  notes  on  the  following  matrices: 

•  where  possible,  direct  statements  were  taken  from  supporting  documents.  No  effort  was  made  to  re¬ 
fine  existing  metrics; 

•  goals  are  often  not  explicitly  stated.  In  that  case,  goal  statements  were  extracted. 

•  the  documentation  [1  and  4]  does  not  state  how  these  goals  are  supported  (i.e.,  the  CARDS  Program 
processes).  The  intersection  in  the  following  matrices  shows  a  simple  “X"  where  processes  should  be. 

B.l  DoD/ CARDS  Goals 

DoD  Software  Reuse  Vision  and  Strategy  (July  15, 1992): 
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'‘Interoperability  should  provide  the  ability  to  locate  and  share  reusable  [assets]  across  domains  and 
among  savices.  Utilize  evolving  technology  to  provide  a  network  of  interconnected  reuse  library  sys¬ 
tems  to  support  capture,  storage,  and  reuse  of  [assets]  within  and  across  specific  domains" 

CARDS  Interoperability  Goals  (extracted): 

Use  US,  the  Ubrary  Interoperability  Service,  to  provide  interoperability  services  to  CARDS  users  (by 
giving  them  access  to  remote  assets)  aud  to  users  of  cooperating  libraries  (by  providing  them  access  to 
CARDS  assets). 

The  DoD  Ooals/CARDS  Goals  Matrix  will  look  as  follows: 


\CARDS 

\Goals 

DoD 

Goals 

Interop,  to 

CARDS  users 

Interop,  to 
cooperating 

libraries 

1 

ri 

locate,  share 

reusable 

assets 

X 

X 

Use  evolving 
technology 

X 

_ 

Figure  B-2  DoD  Goais/Cards  Goals 
All  CARDS  Goals  are  catriod  forward  to  the  next  matrix. 

B.2  Interoperability  Project  Goais 
Provide  automated  access  to  remote  assets  for  CARDS  users,  including: 
retrieve  and  display  the  abstract  of  a  remote  asset: 
retrieve  and  display  thi:  description  of  a  remote  asset;  and 
retrieve  the  contents  of  a  remote  asset  and  store  them  in  the  users  library. 

Provide  automated  access  to  CARDS  assets  for  users  of  cooperating  libraries,  including: 
provide  the  abstract  of  the  CARDS  asset; 
provide  the  description  of  the  CARDS  asset;  and 
provide  the  contents  of  a  CARDS  asset. 
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Collect  usage  and  performance  metrics  to  assess  the  impact  of  interoperation  on  their  respective  library 
operations. 

The  CARDS^Project  Goals  matrix  will  look  as  follows: 


Figure  B*3  CAItDS  Goalt/Project  Goals 

i\Tote  that  “collect  motrlcs*’  is  not  tied  directly  to  any  CARDS  goal.  This  goal  could  be  restated  as  “im¬ 
plement  an  efQdent  client/server  mechanism”  which  would  require  metrics  collection  to  demonstrate 
efficiency.  Stating  the  goal  this  way  also  gives  a  tie  back  to  why  metrics  are  collected. 

B  J  User  Goals 

Assets  exported  from  CARDS: 

Service  Provided:  msg_driver_motif:  BB_PRISM_msg_gcn;  PRISM_MTV;  PP;  XSpread. 

Assets  imported  from  ASSET: 

Service  Provided:  Ada/SQL  Bindings;  OPTIMIZATION  AND  PLANNING  TOOLS;  REUSABLE 
IMAGE  PROCESSING  PACKAGES 

Assets  imported  from  DSRS; 

Service  Provided:  Saeen_And_Data_Manager,  Generic_Report_Handler,  SafeJO;  String_Utilities 
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Figure  B*4  Project  Gk>als/User  Goals 

Projea  fflanagement’s  goal  of  collecting  usage  and  performance  metrics  is  not  reflected  in  any  user 
goal,  and  thus  no  products  or  services  are  found  at  these  intersections.  If  this  goal  was  restated  as  “im* 
plement  an  efficient  client/server  mechanism,”  the  intersections  would  show  such  products  as  TCP/IP 
or  US  client/server.  Measurements  could  then  be  collected  on  these  products/services.  For  the  purpose 
of  this  discussion,  all  goals  will  be  carried  forward  to  the  next  matrix. 


B.4  Metric  question 

•  What  is  the  current  composition  of  a  library’s  set  of  available  extractable  assets? 

•  How  many  searches  are  performed  at  the  library  and  how  many  hits  result  horn  these  searches. 

•  How  many  requests  are  made  to  browse  abstracts  for  assets  in  the  library?  What  is  the  efficiency  of 
accessing  the  requested  abstracts? 

•  How  many  requests  are  made  to  extract  an  asset  from  the  library? 


The  only  performance  metric  question  asked  which  refers  back  to  the  process  steps  used  to  access  the 
library  is  on  the  “efficiency  of  accessing  abstracts."  (After  discussing  this  with  the  Ptoject  Lead,  this 
may  be  a  typo  In  the  original  document  [CARDS04],)  Additional  performance  questions  should  be 
asked.  Detailing  the  process  steps  will  allow  us  to  get  at  these  additional  metric  questions. 
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The  Metric  Ooals/Metric  Questioas  matrix  will  look  like  this: 


\Metrics 

N.  Quest. 
UsctVC 
RroducK. 
Goal  X 

current 

composition 

how  many 
seardies 

bow  many 
hits 

how  many 
requests 
for  abstracts 

efficiency  of 

accessing 

al»tracts 

#of  requests 
to  extract 
asset 

ISSaH 

X 

X 

X 

X 

X 

Interop  to 
CARDS 

X 

X 

Interop  to 
coop. libs 

X 

X 

exported  by 
CARDS 

X 

X 

X 

imported 
from  ASSET 

X 

X 

X 

imported 
from  DSRS 

X 

X 

X 

Figure  B«5  Metric  Goais/Metric  Questions 

B.5  Metrics 

for  available  assets: 
number  of  local  extractable  assets 
number  of  remote  extractable  assets  (from  each  library) 
total  number  of  extractable  assets 


for  searches: 
total  number  of  searches 
number  of  non>null  searches 
number  of  null  searches 
number  of  local  hits 
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number  of  remote  hits 


for  abstract  requests: 
number  of  local  abstract  request 
number  of  remote  abstract  requests  (from  each  library) 
number  of  failures 

average,  min,  and  max  abstract  transfer  times 
average,  min  and  max  abstract  size 


for  asset  requests: 
number  of  local  asset  requests 
number  of  remote  asset  requests  (ftom  each  library) 
number  of  failures 

avg,  min,  and  max  asset  transfer  times 
avg,  min,  and  max  asset  size 
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Metrics  Question/Metiics  matrix,  corn’d: 


Figure  B-6  Metric  Queation/Metric 

Note  that  several  metrics  are  collected  that  have  no  corresponding  metric  questions.  This  matrix  indi* 
cates  which  set  of  metrics  are  needed  to  answer  a  particular  question.  It  also  shows  that  otha  questions 
could  be  answered,  such  as  "what  is  the  percentage  of  failures",  or  "is  the  trend  to  access  assets  Increas¬ 
ing  over  time?" 
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